Papers with pacing-based sampling
Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning (2026.eacl-long)
Copied to clipboard
| Challenge: | Curriculum learning has improved efficiency across machine learning domains, but remains underexplored for language model pretraining. |
| Approach: | They present a systematic investigation of curriculum learning in LLM pretraining . they use vanilla curriculum learning, pacing-based sampling, and interleaved curricula . |
| Outcome: | The proposed framework accelerates convergence in early and mid-training phases, reducing training steps by 18-45% to reach baseline performance. |